Software application providers, including providers of software services and data management services, typically develop applications to collect and manage data from users. It is desirable to test such applications before being released to the users, and to fix any application defects that, for example, cause incorrect or unintended results. Software testing is traditionally the process of executing an application with the intent of finding errors. An application can be tested manually, in which a tester interacts with an application to perform steps of a test case and the tester monitors the results for defects. However, manual testing of an application is generally time consuming and may not cover all test cases or execution paths through an application. Thus, to improve the coverage of testing on an application prior to release, test automation, in which a test script of instructions is created from a test case, may be used. The test scripts are executed repeatedly by testing tools and comparisons are made between actual outcomes and predicted outcomes. Test automation is considered advantageous in traditional data management environments because the test scripts, after being created, can be repetitively executed.
However, in current software application development environments, it is also considered to be burdensome to create test scripts. Not only is it time consuming to create test cases, from which test scripts are derived, but it is also onerous to maintain test scripts after the test scripts are created. For example, when testing an application under development, the ability to quickly react to ever-changing application functionality and requirements is critical. Accordingly, although the benefits of test automation are well known, automated testing in current data management systems is often unutilized and neglected due to the time and costs of test script creation and maintenance.
Furthermore, with the advancement in the complexity of applications, test script creation has become even more difficult in current data management environments. Traditional testing of complex applications to achieve software quality is known to be labor intensive, time consuming, and limited in thoroughness. For example, it is thought that for complex applications, software testing traditionally consumes half of a project's resources in terms of both cost and time. The larger an application, the more complexity there is in terms of both static elements and dynamic elements. Software testing can be considered a tradeoff between budget, time, and quality, and it is considered cost prohibitive to increase test team size to match the growth rate of the complexity of applications. Yet, application defects can be buried within software code and can remain hidden until activated by users. Such activation by users may occur due to real world timing, unpredictable infrastructure effects, and random human interactions with unpredictable input parameters to the applications. Further complications result from the dynamic nature of applications that undergo changes during the software lifecycle, which often invalidates prior test cases and test scripts.
Applications are designed, built, and deployed by teams of fallible engineers, giving applications a fragile characteristic. Yet, due to the fallibility of such teams comprehending the complexity of applications, detecting all or most of the different software failure modes for an application is generally considered infeasible. For example, an application with minimal complexity of just one hundred lines of code may have up to 1014 possible paths that can be executed. Even the most highly skilled tester cannot come up with all such possibilities for testing. Nevertheless, with the ubiquitous nature of complex applications, business and societal costs of big software failures can be astronomical. For example, a failure of a complex application can have such a severe impact on users that the failure can lead to real-life disasters, financial impacts, and so on.
Consequently, what is needed is a system and method that solves the long standing technical problem in the data management arts of state based intelligent test generation that evaluates attributes and capabilities of an application to determine that it meets the desired results through quality assurance, verification, and validation.